{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据加值宣言\n",
    "本项目产出通过selemiun按照中国知网专业检索中主题+关键字+题名+年份结合搜索，挖掘出关于当今人们对于数字经济、数字化转型等50页2150篇论文的作者、题名、关键字、摘要、来源、机构、年份等数据。以解决NPC就业对于数字经济等知识的需求及相关行业的就业分析提供参考的问题。<br/>\n",
    "\n",
    "## MVP数据加值\n",
    "本项目产品的核心是用户浏览从中国知网上爬取的关键字、摘要、来源等论文数据，解决用户对于数字经济+数字化转型+人工智能+工业互联网+大数据产业+一带一路+区块链+制造业+企业管理+数字中国+高质量发展+网络安全相关信息的快速了解并提供一个客观的参考。\n",
    "## 挖掘Query参数\n",
    "(主题=数字经济+数字化转型+人工智能+工业互联网+大数据产业+一带一路+区块链+制造业+企业管理+数字中国+高质量发展+网络安全) AND (关键字=数字经济+数字化转型+人工智能+工业互联网+大数据产业+一带一路+区块链+制造业+企业管理+数字中国+高质量发展+网络安全) AND (题名=数字经济+数字化转型+人工智能+工业互联网+大数据产业+一带一路+区块链+制造业+企业管理+数字中国+高质量发展+网络安全) AND (年份=2020-2018)\n",
    "以上参数可以足够地提供用户对于相关行业领域、价值信息、最新动态等信息地获取，帮助用户节省了时间精力。\n",
    "<br/>\n",
    "\n",
    "分析说明：以上为参数设置，为了解决NPC的问题，参数关键词设置为针对于数字经济的相关词汇最为关键字设置，并将数字经济也设置为主题、题名进行搜索，获取最新的论文观点，将年份设置为2018-2020年。\n",
    "## 思路方法及具体执行\n",
    "**方法选择**：通过使用selenium模拟点击，成功采集中国知网的50页数据；该网站数据在xhr+js+css中,其中值得注意的是在进行参数录入并检索的时候，文章数据是存在一个iframe的标签里的，所以要下载到知网文章数据，就必须先进去iframe标签里，再寻找它的xpath，否则则报错找不到xpath。<br/>\n",
    "\n",
    "**单页数据**：该网站文章下载是规定500篇下载一次，故每次500篇下载一次excel、doc、txt，但500篇-1000篇的过程中，会遇到验证码的验证，这里通过调用百度智能云平台的OCR文字识别（高精度版）识别验证码，并放进验证码框。<br/>\n",
    "\n",
    "**多页数据**：这里爬取了50页2500篇文章，以excel、txt、doc三种形式进行导出。<br/>\n",
    "\n",
    "**系统设计思维**：这里选择了使用selenium进行数据的爬取，但在爬取过程中，主要会遇到两个问题，一个是可能会遇到中国知网链接不是私密链接这个问题（可能是本人电脑问题），本人在这里使用的解决办法是直接selenium点击相信该网站；第二个问题是爬取数据大于500篇后，会遇到验证码界面，调用OCR文字识别解决。<br/>\n",
    "\n",
    "**数据导出**：中国知网本身就又数据导出的控件，只需要通过使用selenium模拟点击下载即可。<br/>\n",
    "\n",
    "**数据整理**：中国知网本身就是庞大的数据库，考虑到用户使用网站时，从海量数据里浏览数据时的杂乱性。故知网将导出的数据进行系统整理，所以这里并没有也不需要对数据进行整理\n",
    "## 心得总结及感谢\n",
    "在爬取数据的过程中，会因为浏览器的原因，显示该网页不是私密链接，经搜查虽然有提供方法，但对于本浏览器并未起到作用，始终未能跳过不是私密链接这个报错，最终发现原来可以直接通过selemiun定位到xpath并模拟点击控件进入到网页中，这一点向大家分享。对于本项目的总结就是：代码编写过程中要做到心细不能急，不要害怕报错，报错是常有的，但是解决报错过程中，保持冷静乐观的心态，解决完一个报错，你就有一份收获。最后感谢老师一整学期的教学与指导，同学们的答疑与帮助。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Final_web_中国知网"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\anaconda\\lib\\site-packages\\ipykernel_launcher.py:21: DeprecationWarning: use options instead of chrome_options\n"
     ]
    }
   ],
   "source": [
    "from selenium import webdriver\n",
    "from selenium.webdriver.common.desired_capabilities import DesiredCapabilities\n",
    "\n",
    "\n",
    "#caps=dict()\n",
    "#caps[\"pageLoadStrategy\"] = \"none\"   # Do not wait for full page load\n",
    "\n",
    "opts = webdriver.ChromeOptions()\n",
    "opts.add_argument('--no-sandbox')#解决DevToolsActivePort文件不存在的报错\n",
    "opts.add_argument('window-size=1920x3000') #指定浏览器分辨率\n",
    "opts.add_argument('--disable-gpu') #谷歌文档提到需要加上一这个属性来规避bug\n",
    "opts.add_argument('--hide-scrollbars') #隐藏滚动条, 应对些特殊页面\n",
    "#opts.add_argument('blink-settings=imagesEnabled=false') #不加载图片, 提升速度\n",
    "#opts.add_argument('--headless') #浏览器不提供可视化页面. linux下如果系统不支持可视化不加这条会启动失败\n",
    "\n",
    "opts.binary_location = r\"C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe\" #\"H:\\_coding_\\Gitee\\InternetNewMedia\\CapstonePrj2016\\chromedriver.exe\"  \n",
    "\n",
    "# \"H:\\_coding_\\Gitee\\InternetNewMedia\\CapstonePrj2016\\chromedriver.exe\"  \n",
    "options = webdriver.ChromeOptions()\n",
    "options.add_argument('--ignore-certificate-errors')\n",
    "driver = webdriver.Chrome( chrome_options = opts) #desired_capabilities=caps,\n",
    "\n",
    "driver.get(\"https://fsso.cnki.net/\")\n",
    "\n",
    "fn={\"output\":{\"raw_data\":\"data\"}}\n",
    "\n",
    "school={\"school\":\"中山大学南方学院\"}\n",
    "driver.find_element_by_xpath('//div[@class=\"submit_input\"]/input').click()\n",
    "driver.find_element_by_xpath('//input[@id=\"o\"]').clear()\n",
    "driver.find_element_by_xpath('//input[@id=\"o\"]').send_keys(school['school'])\n",
    "driver.find_element_by_xpath('//div[@class=\"submit_button\"]').click()\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['CDwindow-9BABCEA3420F4F69366F7A7E7AF5B36A', 'CDwindow-580EB38C0BD2417904D83E0DF6814328']\n",
      "\n"
     ]
    }
   ],
   "source": [
    "\n",
    "driver.find_element_by_xpath('//button[@id=\"details-button\"]').click()#这两行为不是私密链接时使用，若正常访问则删除\n",
    "driver.find_element_by_xpath('//a[@id=\"proceed-link\"]').click()\n",
    "\n",
    "#输入账号密码\n",
    "payload =  {\"account\": \"183014132\", \"password\": \"11101720\"}\n",
    "driver.find_element_by_xpath('//input[@id=\"username\"]').send_keys(payload['account'])\n",
    "driver.find_element_by_xpath('//input[@id=\"password\"]').send_keys(payload['password'])\n",
    "\n",
    "#登录\n",
    "driver.find_element_by_xpath('//button[@class=\"form-element form-button\"]').click()\n",
    "\n",
    "#接受\n",
    "driver.find_element_by_xpath('//input[@name=\"_eventId_proceed\"]').click()\n",
    "\n",
    "#高级检索\n",
    "driver.find_element_by_xpath('//a[@id=\"highSearch\"]').click()\n",
    "\n",
    "#这里已经新建一个新窗口，所以必须进入到新窗口再进行xpath的专业检索\n",
    "print (driver.window_handles)\n",
    "driver.switch_to.window(driver.window_handles[1])  #进入到新窗口\n",
    "\n",
    "# 专业检索\n",
    "driver.find_element_by_xpath('//li[@id=\"1_3\"]/a').click()\n",
    "\n",
    "#关键词输入\n",
    "query=\"((SU=数字经济+数字化转型+人工智能+工业互联网+大数据产业+一带一路+区块链+制造业+企业管理+数字中国+高质量发展+网络安全) AND (KY=数字经济+数字化转型+人工智能+工业互联网+大数据产业+一带一路+区块链+制造业+企业管理+数字中国+高质量发展+网络安全) AND (TI=数字经济+数字化转型+人工智能+工业互联网+大数据产业+一带一路+区块链+制造业+企业管理+数字中国+高质量发展+网络安全) AND (YE=2020-2018))\"\n",
    "driver.find_element_by_xpath('//textarea[@id=\"expertvalue\"]').clear()\n",
    "driver.find_element_by_xpath('//textarea[@id=\"expertvalue\"]').send_keys(query)\n",
    "\n",
    "#点击检索\n",
    "element=driver.find_element_by_xpath('//input[@id=\"btnSearch\"]')\n",
    "main_content = element.get_attribute('innerHTML') #获取到整个页面的内容\n",
    "print(main_content)\n",
    "element.click()  #无显示正常，因为下面的文章页面是html的子页面，所以必须进入到子页面里才可以获取到内容\n",
    "\n",
    "#选择到子页面\n",
    "driver.switch_to.default_content();\n",
    "\n",
    "element = driver.find_element_by_xpath('//html')  #这里的xpath是iframe标签下的整个html\n",
    "main_content = element.get_attribute('innerHTML')  #获取整个页面的内容\n",
    "\n",
    "# 进入iframe 使用ID\n",
    "driver.switch_to.frame(\"iframeResult\")\n",
    "\n",
    "# 获取到table的整个页面的内容\n",
    "#element = driver.find_element_by_xpath('//html//table')\n",
    "\n",
    "driver.switch_to.default_content();\n",
    "driver.switch_to.frame(\"iframeResult\")  #进入到iframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 获取到table的整个页面的内容\n",
    "element = driver.find_element_by_xpath('//html//table')\n",
    "\n",
    "driver.switch_to.default_content();\n",
    "driver.switch_to.frame(\"iframeResult\")  #进入到iframe\n",
    "\n",
    "# 1、点击中文文献\n",
    "driver.find_element_by_xpath('//a[text()=\"中文文献\"]').click()\n",
    "\n",
    "# 2、点击每页显示50\n",
    "driver.find_element_by_xpath('//div[@id=\"id_grid_display_num\"]/a[3]').click()\n",
    "\n",
    "# 3、点击按发表时间排列\n",
    "driver.find_element_by_xpath('//a[text()=\"发表时间\"]').click()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'refresh_token': '25.61f47a7233a2fe191d61029ac2241ee6.315360000.1910393902.282335-20284632', 'expires_in': 2592000, 'session_key': '9mzdDx3NN/N4fIgYoBYWWWEkG823kyA/mtVDXhL+wAmm/qQLBQ3OkXtF4mLS9y6mFpP4A7EB9vekGYVSJYfn5ltDOFtDCQ==', 'access_token': '24.4923a9eb80c91b49f4d4715ec9287b07.2592000.1597625902.282335-20284632', 'scope': 'public vis-ocr_ocr brain_ocr_scope brain_ocr_general brain_ocr_general_basic vis-ocr_business_license brain_ocr_webimage brain_all_scope brain_ocr_idcard brain_ocr_driving_license brain_ocr_vehicle_license vis-ocr_plate_number brain_solution brain_ocr_plate_number brain_ocr_accurate brain_ocr_accurate_basic brain_ocr_receipt brain_ocr_business_license brain_solution_iocr brain_qrcode brain_ocr_handwriting brain_ocr_passport brain_ocr_vat_invoice brain_numbers brain_ocr_business_card brain_ocr_train_ticket brain_ocr_taxi_receipt vis-ocr_household_register vis-ocr_vis-classify_birth_certificate vis-ocr_台湾通行证 vis-ocr_港澳通行证 vis-ocr_机动车购车发票识别 vis-ocr_机动车检验合格证识别 vis-ocr_车辆vin码识别 vis-ocr_定额发票识别 vis-ocr_保单识别 vis-ocr_行程单识别 brain_ocr_vin brain_ocr_quota_invoice brain_ocr_birth_certificate brain_ocr_household_register brain_ocr_HK_Macau_pass brain_ocr_taiwan_pass brain_ocr_vehicle_invoice brain_ocr_vehicle_certificate brain_ocr_air_ticket brain_ocr_insurance_doc brain_formula brain_ocr_webimage_loc wise_adapt lebo_resource_base lightservice_public hetu_basic lightcms_map_poi kaidian_kaidian ApsMisTest_Test权限 vis-classify_flower lpq_开放 cop_helloScope ApsMis_fangdi_permission smartapp_snsapi_base iop_autocar oauth_tp_app smartapp_smart_game_openapi oauth_sessionkey smartapp_swanid_verify smartapp_opensource_openapi smartapp_opensource_recapi fake_face_detect_开放Scope vis-ocr_虚拟人物助理 idl-video_虚拟人物助理', 'session_secret': '3a190c84168de9337b7b826ed3a00019'}\n"
     ]
    }
   ],
   "source": [
    "# encoding:utf-8\n",
    "import requests \n",
    "\n",
    "# client_id 为官网获取的AK， client_secret 为官网获取的SK\n",
    "host = 'https://aip.baidubce.com/oauth/2.0/token?grant_type=client_credentials&client_id=O7HsX7izOe3YpmGixELlo2hG&client_secret=FfozrW5GS2lyx5HoDlwhEIUGVpr6vGG4'\n",
    "response = requests.get(host)\n",
    "if response:\n",
    "    print(response.json())\n",
    "    \n",
    "access_token=response.json()[\"access_token\"]\n",
    "\n",
    "# encoding:utf-8\n",
    "\n",
    "import requests\n",
    "import base64\n",
    "\n",
    "'''\n",
    "通用文字识别\n",
    "'''\n",
    "def outcode(img,access_token):\n",
    "    \"\"\"验证码结果输出\"\"\"\n",
    "    request_url = \"https://aip.baidubce.com/rest/2.0/ocr/v1/accurate_basic\"\n",
    "    # 二进制方式打开图片文件\n",
    "    #f = open('[本地文件]', 'rb')\n",
    "    #img = base64.b64encode(f.read())\n",
    "    params = {\"image\":img}\n",
    "    request_url = request_url + \"?access_token=\"+ access_token\n",
    "    headers = {'content-type': 'application/x-www-form-urlencoded'}\n",
    "    response = requests.post(request_url, data=params, headers=headers)\n",
    "    if len(response.json()[\"words_result\"])==0:\n",
    "        crack_code=123456\n",
    "    else:\n",
    "        crack_code=response.json()[\"words_result\"][0][\"words\"]\n",
    "    return crack_code\n",
    "\n",
    "\n",
    "def isElementExist(element):\n",
    "    \"\"\"验证是否为XXX页面\"\"\"\n",
    "    flag= True\n",
    "    try:\n",
    "        driver.find_element_by_xpath(element)\n",
    "        return flag\n",
    "    except:\n",
    "        flag=False\n",
    "        return flag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "是否为验证码页 False\n",
      "0\n",
      "是否为验证码页 False\n",
      "50\n",
      "是否为验证码页 False\n",
      "100\n",
      "是否为验证码页 False\n",
      "150\n",
      "是否为验证码页 False\n",
      "200\n",
      "是否为验证码页 False\n",
      "250\n",
      "是否为验证码页 False\n",
      "300\n",
      "是否为验证码页 False\n",
      "350\n",
      "是否为验证码页 False\n",
      "400\n",
      "是否为验证码页 False\n",
      "450\n",
      "是否为验证码页 False\n",
      "500\n",
      "是否为验证码页 False\n",
      "0\n",
      "是否为验证码页 False\n",
      "50\n",
      "是否为验证码页 False\n",
      "100\n",
      "是否为验证码页 False\n",
      "150\n",
      "是否为验证码页 False\n",
      "200\n",
      "是否为验证码页 False\n",
      "250\n",
      "是否为验证码页 True\n",
      "True\n",
      "True\n",
      "True\n",
      "True\n",
      "True\n",
      "True\n",
      "False\n",
      "是否为验证码页 False\n",
      "300\n",
      "是否为验证码页 False\n",
      "350\n",
      "是否为验证码页 False\n",
      "400\n",
      "是否为验证码页 False\n",
      "450\n",
      "是否为验证码页 False\n",
      "500\n",
      "是否为验证码页 False\n",
      "0\n",
      "是否为验证码页 False\n",
      "50\n",
      "是否为验证码页 False\n",
      "100\n",
      "是否为验证码页 False\n",
      "150\n",
      "是否为验证码页 False\n",
      "200\n",
      "是否为验证码页 False\n",
      "250\n",
      "是否为验证码页 False\n",
      "300\n",
      "是否为验证码页 False\n",
      "350\n",
      "是否为验证码页 False\n",
      "400\n",
      "是否为验证码页 False\n",
      "450\n",
      "是否为验证码页 False\n",
      "500\n",
      "是否为验证码页 False\n",
      "0\n",
      "是否为验证码页 True\n",
      "True\n",
      "True\n",
      "False\n",
      "是否为验证码页 False\n",
      "50\n",
      "是否为验证码页 False\n",
      "100\n",
      "是否为验证码页 False\n",
      "150\n",
      "是否为验证码页 False\n",
      "150\n",
      "是否为验证码页 False\n",
      "200\n",
      "是否为验证码页 False\n",
      "250\n",
      "是否为验证码页 False\n",
      "300\n",
      "是否为验证码页 False\n",
      "350\n",
      "是否为验证码页 False\n",
      "400\n",
      "是否为验证码页 False\n",
      "450\n",
      "是否为验证码页 False\n",
      "500\n",
      "是否为验证码页 False\n",
      "0\n",
      "是否为验证码页 False\n",
      "50\n",
      "是否为验证码页 False\n",
      "100\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import requests_html\n",
    "import math\n",
    "from PIL import Image\n",
    "import time\n",
    "from random import random\n",
    "element = driver.find_element_by_xpath('//html')\n",
    "main_content =element.get_attribute('outerHTML')  \n",
    "html = requests_html.HTML(html= main_content, url='https://localhost/')\n",
    "content = html.xpath('//*[@id=\"ctl00\"]/table/tbody/tr[3]/td/table/tbody/tr/td/div/div')\n",
    "\n",
    "for page in range(0,50):\n",
    "    flag=isElementExist('//img[@id=\"CheckCodeImg\"]')#调用函数，判断是否为验证码页\n",
    "    print(\"是否为验证码页\",flag)\n",
    "    if flag:\n",
    "        while True:\n",
    "            flag=isElementExist('//img[@id=\"CheckCodeImg\"]')\n",
    "            print(flag)\n",
    "            if flag:\n",
    "                ele=driver.find_element_by_id('CheckCodeImg')#如果是，保存验证码图片\n",
    "                ele.screenshot('D:/code.png')\n",
    "                Image.open('D:/code.png')\n",
    "                f=open('D:/code.png','rb')\n",
    "                img=base64.b64encode(f.read())\n",
    "                #调用百度文字识别API，识别验证码\n",
    "                crack_code=outcode(img,access_token)\n",
    "                #填写验证码\n",
    "                driver.switch_to.default_content()\n",
    "                driver.switch_to.frame(\"iframeResult\")\n",
    "                #clear_and_send('//input[id=\"CheckCode\"]',crack_code,driver=driver)\n",
    "                driver.find_element_by_xpath('//input[@id=\"CheckCode\"]').send_keys(crack_code)\n",
    "                driver.find_element_by_xpath('//input[@type=\"button\"]').click()\n",
    "                flag_02=isElementExist('//img[@id=\"CheckCodeImg\"]')\n",
    "                if flag_02 ==False:\n",
    "                    continue\n",
    "            else:\n",
    "                break\n",
    "        continue\n",
    "    element = driver.find_element_by_xpath('//html')\n",
    "    main_content =element.get_attribute('outerHTML')  \n",
    "    html = requests_html.HTML(html= main_content, url='https://localhost/')\n",
    "    已选文献 = html.xpath('//table[@id=\"J_ORDER\"]/tbody/tr[2]/td/table/tbody/tr/td[1]/div/span/em')[0].lxml.text_content()\n",
    "    print(int(已选文献))\n",
    "    if int(已选文献) < 500:\n",
    "        # 勾选本页文献\n",
    "        driver.find_element_by_xpath('//input[@id=\"selectCheckbox\"]').click()\n",
    "        # 等待时间 平均时间38*103\n",
    "        time.sleep(3)\n",
    "        # 翻页\n",
    "        driver.switch_to.default_content()\n",
    "        driver.switch_to.frame(\"iframeResult\")\n",
    "        element = driver.find_element_by_xpath('//html')\n",
    "        main_content =element.get_attribute('outerHTML')  \n",
    "        main_content\n",
    "        a = driver.find_element_by_xpath('//a[text()=\"下一页\"]')\n",
    "        a.click()\n",
    "\n",
    "    else:\n",
    "        # 进入导出/参考文献  页面（不同窗口）\n",
    "        driver.switch_to.default_content()\n",
    "        driver.switch_to.frame(\"iframeResult\")\n",
    "        element = driver.find_element_by_xpath('//html')\n",
    "        main_content =element.get_attribute('outerHTML') \n",
    "        main_content\n",
    "        a = driver.find_element_by_xpath('//a[text()=\" 导出/参考文献\"]')\n",
    "        a.click()\n",
    "        driver.switch_to.window(driver.window_handles[2]) #进入新窗口\n",
    "        driver.find_element_by_xpath('//ul[@id=\"SaveTypeList\"]/li[6]/span/a').click() #refworks\n",
    "        driver.find_element_by_xpath('//input[@id=\"exportTxt\"]').click()  #txt     \n",
    "        driver.find_element_by_xpath('//input[@id=\"exportExcel\"]').click() #excel\n",
    "        driver.find_element_by_xpath('//input[@id=\"exportWord\"]').click()#doc\n",
    "\n",
    "        # 清除已选文章，继续翻页\n",
    "        # 回到cnki搜索文章页面（切换窗口）\n",
    "        driver.switch_to.window(driver.window_handles[1])\n",
    "        # 清除已选\n",
    "        driver.switch_to.default_content()\n",
    "        driver.switch_to.frame(\"iframeResult\")\n",
    "        element = driver.find_element_by_xpath('//html')\n",
    "        main_content =element.get_attribute('outerHTML') \n",
    "        a = driver.find_element_by_xpath('//a[text()=\"清除\"]').click()\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
